🔤 Language Tokenizers - abnv · Scour

Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models

arxiv.org·14h

Reducing Tokenization Premiums for Low-Resource Languages

arxiv.org·14h

⚡Tokenizer Benchmarks

LLMs - Custom Tokenizers

dev.to·2d·

Discuss: DEV

⚡Tokenizer Benchmarks

t2x - a CLI tool for AI-first text operations

shruggingface.com·16h

🔄Incremental Lexing

I [[musttail]] You About a Tokenizer

neilhenning.dev·2d

DevUtils, a Mac app for text conversion and processing

blakewatson.com·1d

🔄Incremental Lexing

**Abstract:** This paper presents a novel system, “LexiGuard,” for automated review and anomaly detection within large-scale collaborative LaTeX ...

freederia.com·1d

🔄Incremental Lexing

Coding Alpha: How We Use Python to Scrape Fintech Sentiment in Brazil

velthorneassetmanagement.com·8h·

Discuss: DEV

📋JSON Parsing

Iterative multi-word anagram solver

boulter.com·21h

🤖Suffix Automata

The Only Two Markup Languages

lobste.rs·21h·

Discuss: Lobsters

🎨Domain-Specific Languages

Patterns All the Way Down: A Generalization for Graph-Like Things

medium.com·2h·

Discuss: Hacker News

🪢Rope Data Structures

Rainbow Query Language

rbql.org·20h

📊Query Optimizers

istmarc/tenseur: C++23 Tensor, neural networks and mathematical library

github.com·54m·

Discuss: r/cpp

🦀MIR Optimization

Lab 6 Issue 1: An accessible PDF/MP3/TXT polyglot, Fantasy Footbyte

lab6.com·16h

🏷️Symbol Interning

How Does ChatGPT Work? A Guide for the Rest of Us

producttalk.org

·5h

🚀Tokenizer Performance

Hugging Face Releases FineTranslations, a Trillion-Token Multilingual Parallel Text Dataset

infoq.com·3d

🚀Tokenizer Performance

Making an LSP for great good

thunderseethe.dev·19h·

Discuss: Lobsters, Hacker News

docs.gitlab.com·22h

↔️Bidirectional Sync

Exploring Text Compression

denvaar.dev·16h

📦Compression Algorithms

dev.to·6h·

Discuss: DEV

🧠Semantic Parsing

Loading more...